With the increased popularity of DVDs, music delivery over the Internet, and digital cameras, digital media have become commonplace. Engineers use a variety of techniques to process digital audio, video, and images efficiently while still maintaining quality. To understand these techniques, it helps to understand how the audio, video, and image information is represented and processed in a computer.
I. Representation of Media Information in a Computer
A computer processes media information as a series of numbers representing that information. For example, a single number may represent the intensity of brightness or the intensity of a color component such as red, green or blue for each elementary small region of a picture, so that the digital representation of the picture consists of one or more arrays of such numbers. Each such number may be referred to as a sample. For a color image, it is conventional to use more than one sample to represent the color of each elemental region, and typically three samples are used. The set of these samples for an elemental region may be referred to as a pixel, where the word “pixel” is a contraction referring to the concept of a “picture element.” For example, one pixel may consist of three samples that represent the intensity of red, green and blue light necessary to represent the elemental region. Such a pixel type is referred to as an RGB pixel. Several factors affect quality, including sample depth, resolution, and frame rate (for video).
Sample depth is a property normally measured in bits that indicates the range of numbers that can be used to represent a sample. When more values are possible for the sample, quality can be higher because the number can capture more subtle variations in intensity and/or a greater range of values. Images with higher resolution tend to look crisper than other images and contain more discernable useful details. Video with higher frame rate tends to mimic the smooth motion of natural objects better than other video, and can similarly be considered to contain more detail in the temporal dimension. For all of these factors, the tradeoff for high quality is the cost of storing and transmitting the information in terms of the bit rate necessary to represent the sample depth, resolution and frame rate, as Table 1 shows.
TABLE 1Bit rates for different quality levels of raw videoBits Per PixelResolutionFrame RateBit Rate(sample depth times(in pixels,(in frames(in millions ofsamples per pixel)Width × Height)per second)bits per second)8 (value 0-255,160 × 1207.51.2monochrome)24 (value 0-255, RGB)320 × 2401527.624 (value 0-255, RGB)640 × 48030221.224 (value 0-255, RGB)1280 × 720 601327.1
Despite the high bit rate necessary for sending high quality video (such as HDTV), companies and consumers increasingly depend on computers to create, distribute, and play back high quality content. For this reason, engineers use compression (also called source coding or source encoding) to reduce the bit rate of digital media. Compression decreases the cost of storing and transmitting the information by converting the information into a lower bit rate form. Decompression (also called decoding) reconstructs a version of the original information from the compressed form. A “codec” is an encoder/decoder system. Two categories of compression are lossless compression and lossy compression.
Lossless compression reduces the bit rate of information by removing redundancy from the information without any reduction in fidelity. For example, a series often consecutive pixels that are all exactly the same shade of red could be represented as a code for the particular shade of red and the number ten as a “run length” of consecutive pixels, and this series can be perfectly reconstructed by decompression from the code for the shade of red and the indicated number (ten) of consecutive pixels having that shade of red. Lossless compression techniques reduce bit rate at no cost to quality, but can only reduce bit rate up to a certain point. Decreases in bit rate are limited by the inherent amount of variability in the statistical characterization of the input data, which is referred to as the source entropy. Entropy coding is another term for lossless compression.
In contrast, with lossy compression, the quality suffers somewhat but the achievable decrease in bit rate is more dramatic. For example, a series of ten pixels, each being a slightly different shade of red, can be approximated as ten pixels with exactly the same particular approximate red color. Lossy compression techniques can be used to reduce bit rate more than lossless compression techniques, but some of the reduction in bit rate is achieved by reducing quality, and the lost quality cannot be completely recovered. Lossy compression is often used in conjunction with lossless compression—in a system design in which the lossy compression establishes an approximation of the information and lossless compression techniques are applied to represent the approximation. For example, the series often pixels, each a slightly different shade of red, can be represented as a code for one particular shade of red and the number ten as a run-length of consecutive pixels. In decompression, the original series would then be reconstructed as ten pixels with the same approximated red color.
II. Quantization
According to one possible definition, quantization is a term used for an approximating non-reversible mapping function commonly used for lossy compression, in which there is a specified set of possible output values, and each member of the set of possible output values has an associated set of input values that result in the selection of that particular output value. A variety of quantization techniques have been developed, including scalar or vector, uniform or non-uniform, and adaptive or non-adaptive quantization.
A. Scalar Quantizers
According to one possible definition, a scalar quantizer is an approximating functional mapping x→Q[x] of an input value x to a quantized value Q[x]. FIG. 1 shows a “staircase” I/O function (100) for a scalar quantizer. The horizontal axis is a number line for a real number input variable x, and the vertical axis indicates the corresponding quantized values Q[x]. The number line is partitioned by thresholds such as the threshold (110). Each value of x within a given range between a pair of adjacent thresholds is assigned the same quantized value Q[x]. For example, each value of x within the range (120) is assigned the same quantized value (130). (At a threshold, one of the two possible quantized values is assigned to an input x, depending on the system.) Overall, the quantized values Q[x] exhibit a discontinuous, staircase pattern. The distance the mapping continues along the number line depends on the system, typically ending after a finite number of thresholds. The placement of the thresholds on the number line may be uniformly spaced (as shown in FIG. 1) or non-uniformly spaced.
A scalar quantizer can be decomposed into two distinct stages. The first stage is the classifier stage, in which a classifier function mapping x→A[x] maps an input x to a quantization index A[x], which is often integer-valued. In essence, the classifier segments an input number line or data set. FIG. 2a shows a generalized classifier (200) and thresholds for a scalar quantizer. As in FIG. 1, a number line for a real number variable x is segmented by thresholds such as the threshold (210). Each value of x within a given range such as the range (220) is assigned the same quantized value Q[x]. FIG. 2b shows a numerical example of a classifier (250) and thresholds for a scalar quantizer.
In the second stage, a reconstructor functional mapping k→β[k] maps each quantization index k to a reconstruction value β[k]. In essence, the reconstructor places steps having a particular height relative to the input number line segments (or selects a subset of data set values) for reconstruction of each region determined by the classifier. The reconstructor functional mapping may be implemented, for example, using a lookup table. Overall, the classifier relates to the reconstructor as follows:Q[x]=β[A[x]]  (1).
The distortion introduced by using such a quantizer may be computed with a difference-based distortion measure d(x−Q[x]). Typically, such a distortion measure has the property that d(x−Q[x]) increases as x−Q[x] deviates from zero; and typically each reconstruction value lies within the range of the corresponding classification region, so that the straight line that would be formed by the functional equation Q[x]=x will pass through every step of the staircase diagram (as shown in FIG. 1) and therefore Q[Q[x]] will typically be equal to Q[x]. In general, a quantizer is considered better in rate-distortion terms if the quantizer results in a lower average value of distortion than other quantizers for a given bit rate of output. More formally, a quantizer is considered better if for a source random variable X, the expected (i.e., the average or statistical mean) value of the distortion measure D=EX{d(X−Q[X])} is lower for an equal or lower entropy H of A[X]. The most commonly-used distortion measure is the squared error distortion measure, for which d(|x−y|)=|x−y|2. When the squared error distortion measure is used, the expected value of the distortion measure (D) is referred to as the mean squared error.
B. Dead Zone+Uniform Threshold Quantizers
According to one possible definition, a dead zone plus uniform threshold quantizer [“DZ+UTQ”] is a quantizer with uniformly spaced threshold values for all classifier regions except the one containing the zero input value (which is called the dead zone [“DZ”]). A DZ+UTQ has a classifier index mapping rule x→A[x] that can be expressed based on two parameters. FIG. 3 shows a staircase I/O function (300) for a DZ+UTQ, and FIG. 4a shows a generalized classifier (400) and thresholds for a DZ+UTQ. The parameter s, which is greater than 0, indicates the step size for all steps other than the DZ. Mathematically, all si are equal to s for i≠0. The parameter z, which is greater than or equal to 0, indicates the ratio of the DZ size to the size of the other steps. Mathematically, s0=z·s. In FIG. 4a, z is 2, so the DZ is twice as wide as the other classification zones. The index mapping rule x→A[x] for a DZ+UTQ can be expressed as:
                                          A            ⁡                          [              x              ]                                =                      sign            ⁢                                                  ⁢                          (              x              )                        *                          max              ⁡                              (                                  0                  ,                                      ⌊                                                                                                                      x                                                                          s                                            -                                              z                        2                                            +                      1                                        ⌋                                                  )                                                    ,                            (        2        )            where └⋅┘ denotes the smallest integer less than or equal to the argument and where sign(x) is the function defined as:
                              sign          ⁡                      (            x            )                          =                  {                                                                                                                +                      1                                        ,                                                                                                                                      for                        ⁢                                                                                                  ⁢                        x                                            ≥                      0                                        ,                                                                                                                                          -                      1                                        ,                                                                                                              for                      ⁢                                                                                          ⁢                      x                                        <                    0.                                                                        .                                              (        3        )            
FIG. 4b shows a numerical example of a classifier (450) and thresholds for a DZ+UTQ with s=1 and z=2. FIGS. 1, 2a, and 2b show a special case DZ+UTQ with z=1. Quantizers of the UTQ form have good performance for a variety of statistical sources. In particular, the DZ+UTQ form is optimal for the statistical random variable source known as the Laplacian source.
In some system designs (not shown), an additional consideration may be necessary to fully characterize a DZ+UTQ classification rule. For practical reasons there may be a need to limit the range of values that can result from the classification function A[x] to some reasonable finite range. This limitation is referred to as clipping. For example, in some such systems the classification rule could more precisely be defined as:
                                          A            ⁡                          [              x              ]                                =                                    sign              ⁡                              (                x                )                                      *                          min              ⁡                              [                                  g                  ,                                      max                    ⁡                                          (                                              0                        ,                                                  ⌊                                                                                                                                                    x                                                                                            s                                                        -                                                          z                              2                                                        +                            1                                                    ⌋                                                                    )                                                                      ]                                                    ,                            (        4        )            where g is a limit on the absolute value of A[x]. In much of the theoretical analysis presented herein, consideration of clipping is omitted as it unduly complicates the analysis without advancing the explanation. Moreover, although the clipping shown in the above example is symmetric about zero, the clipping does not need to be symmetric, and often is not exactly symmetric. For example, a common clipping range would be such that the value of A[x] is limited to some range from −2B to +2B−1 so that A[x] can be represented as an integer using a two's complement representation that uses B+1 bits, where B+1 may be equal to 8 or 16 or another particular selected number of bits.
C. Reconstruction Rules
Different reconstruction rules may be used to determine the reconstruction value for each quantization index. These include the optimal reconstruction rule and the single offset reconstruction rule (of which the mid-point reconstruction rule is an example). FIG. 5 shows reconstruction points according to different reconstruction rules for a particular shape of a source probability distribution function ƒ(x). For a range of values between two thresholds tj and tj+1, the reconstruction value rj,mid according to the mid-point reconstruction rule bisects the range (thus, rj,mid=(tj+tj+1)/2). For the example probability distribution function shown in FIG. 5, this fails to account for the fact that values to the left of the mid-point are more likely than values to the right of the mid-point. The reconstruction value rj,opt according to the optimal reconstruction rule accounts for the probability distribution.
In general, a probability distribution function [“pdf”] indicates the probabilities for the different values of a variable. One possible definition of the optimal reconstruction value rj,opt for each region between two neighboring thresholds tj and tj+1 for a pdf ƒ(x) can be expressed as:
                              r                      j            ,            opt                          =                                            min                              -                1                                      y                    ⁢                                    ∫                              t                j                                            t                                  j                  +                  1                                                      ⁢                                                  ⁢                                          d                ⁡                                  (                                      x                    -                    y                                    )                                            ⁢                              f                ⁡                                  (                  x                  )                                            ⁢              d              ⁢                                                          ⁢                              x                .                                                                        (        5        )            Assuming that the pdf ƒ(x) for a given source is symmetric around zero, one possible definition of the optimal reconstruction rule of a DZ+UTQ for a symmetric, difference-based distortion measure d(|x−y|) is:
                              β          ⁡                      [            k            ]                          =                  {                                                                                                                                                                  min                                                      -                            1                                                                          y                                            ⁢                                                                        ∫                          0                                                      zs                            2                                                                          ⁢                                                                                                  ⁢                                                                              [                                                                                          d                                ⁡                                                                  (                                                                                                                                                x                                      -                                      y                                                                                                                                            )                                                                                            +                                                              d                                ⁡                                                                  (                                                                                                                                                x                                      -                                      y                                                                                                                                            )                                                                                                                      ]                                                    ⁢                                                      f                            ⁡                                                          (                              x                              )                                                                                ⁢                          d                          ⁢                                                                                                          ⁢                          x                                                                                      ,                                                                                                                                      for                        ⁢                                                                                                  ⁢                        k                                            =                      0                                        ,                                                                                                                                                                  sign                        ⁡                                                  (                          k                          )                                                                    ⁢                                                                                          ⁢                                                                                                    min                                                          -                              1                                                                                y                                                ⁢                                                                              ∫                                                                                          zs                                2                                                            +                                                                                                (                                                                                                                                                  k                                                                                                              -                                    1                                                                    )                                                                ⁢                                s                                                                                                                                                    zs                                2                                                            +                                                                                                                                  k                                                                                                  ⁢                                s                                                                                                              ⁢                                                                                                          ⁢                                                                                    d                              ⁡                                                              (                                                                                                                                        x                                    -                                    y                                                                                                                                    )                                                                                      ⁢                                                          f                              ⁡                                                              (                                x                                )                                                                                      ⁢                            d                            ⁢                                                                                                                  ⁢                            x                                                                                                                ,                                                                                                              for                      ⁢                                                                                          ⁢                      k                                        ≠                    0.                                                                        ,                                              (        6        )            where y is the quantized value Q[x], and where the rule finds the quantized value Q[x] that results in the smallest distortion according to the distortion measure. Typically, the optimal quantized value for β[0] is equal to 0, and that will be assumed to be true for the remainder of this description. For minimizing mean squared error, the optimal reconstruction rule sets the reconstruction value for each region equal to the conditional mean of the input values in that region. Stated more precisely, the optimal reconstruction value rj,opt for the region between two neighboring thresholds tj and tj+1 for a pdf ƒ(x) when using the mean squared error distortion measure is given by
                              r                      j            ,            opt                          =                                                            ∫                                  t                  j                                                  t                                      j                    +                    1                                                              ⁢                                                x                  ·                                      f                    ⁡                                          (                      x                      )                                                                      ⁢                                                                  ⁢                d                ⁢                                                                  ⁢                x                                                                    ∫                                  t                  j                                                  t                                      j                    +                    1                                                              ⁢                                                f                  ⁡                                      (                    x                    )                                                  ⁢                                                                  ⁢                d                ⁢                                                                  ⁢                x                                              .                                    (        7        )            
According to one possible definition for a DZ+UTQ, the single-offset reconstruction rule is based on an offset parameter A, where ordinarily 0<Δ≤s/2, and the rule is:
                              β          ⁡                      [            k            ]                          =                  {                                                                                          0                    ,                                                                                                                                      for                        ⁢                                                                                                  ⁢                        k                                            =                      0                                        ,                                                                                                                                                                  sign                        ⁡                                                  (                          k                          )                                                                    ⁡                                              [                                                                                                            (                                                                                                                                  k                                                                                                  +                                                                  z                                  2                                                                -                                1                                                            )                                                        ⁢                            s                                                    +                          Δ                                                ]                                                              ,                                                                                                              for                      ⁢                                                                                          ⁢                      k                                        ≠                    0.                                                                        .                                              (        8        )            
The mid-point reconstruction rule is a special case of the single-offset reconstruction rule, specified by Δ=s/2. Mid-point reconstruction is commonly used for convenience due to its simplicity. And, in the limit as s becomes very small, the performance of the mid-point rule becomes optimal under a variety of well-behaved mathematical conditions.
D. Specifying Reconstruction Values, Constructing Classifiers
Standards and product specifications that focus only on achieving interoperability will often specify reconstruction values without necessarily specifying the classification rule. In other words, some specifications may define the functional mapping k→β[k] without defining the functional mapping x→A[x]. This allows a decoder built to comply with the standard/specification to reconstruct information correctly. In contrast, encoders are often given the freedom to change the classifier in any way that they wish, while still complying with the standard/specification.
Numerous systems for adjusting quantization thresholds have been developed. Many standards and products specify reconstruction values that correspond to a typical mid-point reconstruction rule (e.g., for a typical simple classification rule) for the sake of simplicity. For classification, however, the thresholds can in fact be adjusted so that certain input values will be mapped to more common (and hence, lower bit rate) indices, which makes the reconstruction values closer to optimal. FIG. 6 shows such adjusted thresholds for a classifier (600). The original thresholds (such as old tj) are situated halfway between the reconstruction points. The thresholds are moved outward on the number line, away from 0. Before the adjustment, a marginal value (shown between the old tj and the new tj) is mapped to rj. After the adjustment, the marginal value is mapped to r0. The decoder performs reconstruction without knowledge of the adjustments done in the encoder.
For optimal encoding, an encoder may adjust quantization thresholds to optimally fit a given set of reconstruction values as follows. The probability pj for the source random variable X to fall within a range j between tj and tj+1 (where tj+1>tj) for a source pdf ƒ(x) is:
                                          p            j                    =                                    ∫                              t                j                                            t                                  j                  +                  1                                                      ⁢                                          f                ⁡                                  (                  x                  )                                            ⁢                                                          ⁢              d              ⁢                                                          ⁢              x                                      ,                            (        9        )            and the number of bits necessary to represent an event with probability pj in an ideal lossless communication system may be quantified as:
                                          h            j                    =                                    log              2                        ⁢                          1                              p                j                                                    ,                            (        10        )            where the hj is expressed in terms of bits. The total entropy of the classifier is thengiven by
                    H        =                              ∑            j                                                          ⁢                                          ⁢                                                    p                j                            ·                              h                j                                      ⁢                                                  ⁢                          bits              .                                                          (        11        )            
In general, if the encoder is required to use bj bits to indicate the selection of the reconstruction value rj, the encoder may evaluate and optimize its thresholds according to minimization of the rate-distortion relation D+λR, where D indicates distortion, R indicates bit usage, and λ is a tuning parameter for favoring a particular selected balance between distortion and bit rate. For each particular threshold tj+1 between two points rj and rj+1, the encoder can set tj+1 to the x that satisfies:d(x−rj)+λbj=d(x−rj+1)+λbj+1  (12).
In an ideal design, bj will be approximately equal to hj, and modern lossless coding techniques can be used to very nearly achieve this goal. In a design using some non-ideal lossless coding technique to represent the output of the classifier, bj may have some other value.
Note in summation that optimal decision thresholds can be selected using equation (12), that optimal reconstruction values can be selected using equation (5) or (7), and that optimal bit usage can be computed by setting bj equal to hj as given by equation (10) or to the number of bits used in some other lossless code (such as a Huffman code designed using equation (9) or a fixed-length code). In some highly-optimized scalar quantizer system designs, reconstruction values (initially uniformly spaced) are analyzed to adjust thresholds in encoder analysis, then use of the adjusted thresholds is analyzed to set the number of bits needed to represent the output of the classifier using lossless coding and to set the reconstruction values in decoder analysis. The new reconstruction values are then analyzed to adjust thresholds, and so on, until the thresholds and/or reconstruction values stabilize across iterations.
III. Compression and Decompression Systems
In general, video compression techniques include “intra-picture” compression and “inter-picture” compression, where a picture is, for example, a progressively scanned video frame, an interlaced video frame (having alternating lines for video fields), or an interlaced video field. For progressive frames, intra-picture compression techniques compress individual frames (typically called I-frames or key frames), and inter-picture compression techniques compress frames (typically called predicted frames, P-frames, or B-frames) with reference to preceding and/or following frames (typically called reference or anchor frames).
Both intra and inter-picture compression techniques often use a reversible frequency transform operation, which generates a set of frequency domain (i.e., spectral) coefficients. For intra-picture compression, the transform is typically applied to a block of samples. For inter-picture compression, the transform is typically applied to a block of motion-compensation prediction residual information. A discrete cosine transform [“DCT”] is a type of frequency transform. The resulting blocks of transform coefficients are quantized and entropy encoded. A decoder typically entropy decodes and reconstructs transform coefficients (e.g., DCT coefficients) that were quantized and performs an inverse frequency transform such as an IDCT.
A. Intra-Compression in Windows Media Video, Version 8 [“WMV8”]
Microsoft Corporation's Windows Media Video, Version 8 [“WMV8”] includes a video encoder and a video decoder. The WMV8 encoder uses intra-frame and inter-frame compression, and the WMV8 decoder uses intra-frame and inter-frame decompression.
FIG. 7 illustrates block-based intraframe compression (700) of a 8×8 block (705) of samples in a frame in the WMV8 encoder. The WMV8 encoder here splits a frame into 8×8 blocks of samples and applies an 8×8 DCT (710) to individual blocks such as the block (705). The encoder quantizes (720) the DCT coefficients (715), resulting in an 8×8 block of quantized DCT coefficients (725). For example, the encoder applies a uniform, scalar quantization step size to each coefficient.
Further encoding varies depending on whether a coefficient is a DC coefficient, an AC coefficient in the top row or left column, or another AC coefficient. The encoder encodes the DC coefficient (726) as a differential from the DC coefficient (736) of a neighboring 8×8 block, which is a previously encoded top or left neighbor block. The encoder entropy encodes (740) the differential. The entropy encoder can encode the left column or top row of AC coefficients as differentials from a corresponding column or row of a neighboring 8×8 block. FIG. 7 shows the left column (727) of AC coefficients encoded as differentials (747) from the left column (737) of the neighboring (actually situated to the left) block (735). The encoder scans (750) the 8×8 block (745) of predicted, quantized AC DCT coefficients into a one-dimensional array (755) and then entropy encodes the scanned coefficients using a variation of run length coding (760). The encoder selects an entropy code from one or more run/level/last tables (765) and outputs the entropy code.
A WMV8 decoder (not shown) produces a reconstructed version of the original block (705). The decoder determines the DC predictor for the DC coefficient and decodes the DC differential. In particular, the following pseudocode illustrates the DC differential decoding process in WMV8.
DCDifferential = vlc_decode( )if (DCDifferential == ESCAPECODE)  DCDifferential = flc_decode(8)DCSign = flc_decode(1)if (DCSign == 1)  DCDifferential = −DCDifferential
The WMV8 decoder combines the DC differential with the predictor for the DC coefficient to reconstruct the DC coefficient. The decoder entropy decodes the AC coefficients using one or more run/level/last tables, and scans the coefficients back into a two-dimensional array. The WMV decoder computes a predictor for the top row or left column of AC coefficients if appropriate. The decoder inverse quantizes the coefficients and performs an IDCT.
While DC differential coding and decoding as in WMV8 provide good performance in many scenarios, there are opportunities for improvement. In particular, DC differential coding and decoding as in WMV8 are not easily applied for smaller quantization sizes. This is because at the smaller quantization sizes, VLC code table size for DC differentials becomes inefficiently large for many devices for practical applications.
B. Video Codec Standards
Various standards specify aspects of video decoders as well as formats for compressed video information. These standards include H.261, MPEG-1, H.262 (also called MPEG-2), H.263, and MPEG-4. Directly or by implication, these standards may specify certain encoder details, but other encoder details are not specified. Different standards incorporate different techniques, but each standard typically specifies some kind of inverse frequency transform and entropy decoding. For information, see the respective standard documents.